Flash units are often used to acquire an image of a scene under low-light conditions. However, flash units produce a variety of undesirable effects and artifacts. Objects near the flash unit tend to be over-exposed, while distant objects tend to be under-exposed because the flash intensity decreases as the square of the distance from the camera.
Furthermore, flash units also produce undesirable reflections. Often, one sees the reflection of an object that lies outside the field of view of the camera but is strongly lit by the flash unit, or by a specular object within the field of view. Even more often, one sees strong highlights due to reflections by glossy objects in the scene.
Images acquired of shallow-depth, indoor scenes with flash units can have significantly enhanced details and reduced noise compared to images acquired with just ambient illumination, Eisemann and Durand, “Flash photography enhancement via intrinsic relighting,” ACM Transactions on Graphics 23, 3 (August), 673-678, 2004; and Petschnigg, et al., “Digital photography with flash and no-flash image pairs,” ACM Transactions on Graphics 23, 3 (August), 664-672, 2004. Methods to remove flash shadows, reduce redeye, and perform white balancing are known. Most prior art methods operate on images acquired of indoor scenes with shallow depth ranges. In such cases, the flash unit adequately illuminates most objects in the scene.
It is desired to enhance images of indoor and outdoor scenes with large variations in depth and significant variations in ambient illumination.
Noise in a flash image can actually be higher than that in an ambient image for distant objects. To enhance the ambient image, prior art methods have used variants of a joint bilateral filter.
A number of methods are know for enhancing images using high dynamic range (HDR) images, image gradients, and other techniques. See Mann and Picard, “Being undigital with digital cameras: Extending dynamic range by combining differently exposed pictures,” Proceedings of IS and T 46th annual conference, 422-428, 1995; Debevec and Malik, “Recovering high dynamic range radiance maps from photographs,” Proceedings of the 24th annual conference on Computer graphics and interactive techniques, ACM Press/Addison-Wesley Publishing Co., 369-378, 1997; Fattal, et al., “Gradient Domain High Dynamic Range Compression,” Proceedings of SIGGRAPH 2002, ACM SIGGRAPH, pp. 249-256, 2002; Perez, et al., “Poisson image editing,” Proceedings of SIGGRAPH 2003, pp. 313-318, 2003; Raskar, et al., “Image Fusion for Context Enhancement and Video Surrealism,” Proceedings of NPAR, 2004; Agarwala, et al., “Interactive digital photomontage,” ACM Transactions on Graphics 23, pp. 294-302, August 2004; Sun, et al., “Poisson matting,” ACM Trans. Graph. 23, pp. 315-321, 2004. It should be noted that, in the prior art, HDR images are generally acquired by varying the shutter exposure time for each image.
Some prior art methods remove reflections from flash images by decomposing the flash image into diffuse and specular components using a polarization filter, by changing focus, or by changing viewpoint, Nayar, et al., “Separation of reflection components using color and polarization,” International Journal of Computer Vision 21, pp. 163-186, February 1997; Schechner, et al., “Separation of transparent layers using focus,” International Journal of Computer Vision 39, pp. 25-39, August 2000; Farid and Adelson, “Separating reflections and lighting using independent components analysis,” 1999 Conference on Computer Vision and Pattern Recognition (CVPR 1999), pp. 1262-1267, 1999; and Szeliski, et al., “Layer extraction from multiple images containing reflections and transparency,” 2000 Conference on Computer Vision and Pattern Recognition (CVPR 2000), pp. 2000. A belief propagation based method minimizes the number of edges in a reflection-free decomposition of a single image, Levin, et al., “Separating reflections from a single image using local features,” 2004 Conference on Computer Vision and Pattern Recognition (CVPR 2004), 2004.
On-board sensing and processing allows modern cameras to automatically select the flash power and shutter exposure time setting based on aggregate measurements of scene brightness and distance. For example, the Canon A-TTL camera uses a pre-flash and a photo-sensor sensor on the flash unit to determine the illumination that is needed for the scene. The Nikon-3D camera system uses camera-to-subject distance to focus the lens. This information can also be used to determine the flash power and exposure. It is important to note that in all these cases the selected flash and exposure settings for a particular image do not necessarily ensure that all objects in the scene are adequately illuminated. Furthermore, those settings are entirely based on sensing geometry and ambient lighting in the scene before an image is acquired, and do not consider the actual effect of flash illumination. Also these settings are non-adaptive. These methods do not consider the effect of prior images captured while capturing multiple images for a HDR scene.